The Advantage Learning Operator

Value-based reinforcement learning typically involves the repeated application of an update rule, such as the Bellman operator TB, to an action-value function. Recent work has explored the use of alternative operators, which remain optimality-preserving and may result in improved performance. In this report, I study in particular the advantage learning operator, TALQ = TBQ − α(V − Q). A theoretical analysis of learning as an estimator of Q-value ordering shows that advantage learning may compensate for high Q-learning step sizes. I show further that advantage learning grants increased robustness to the presence of stochasticity, and discuss the importance of committing to current estimates of Q-value ordering, and of increasing action gaps. Finally, I propose two algorithms for on-line optimization of the advantage learning parameter α, demonstrating successful proofs-of-concept in simple MDPs.